Network fault detection and quality of service improvement systems and methods

ABSTRACT

A Quality of Service (QoS) improvement system and method for transport network fault detection and QoS improvement so that a telecommunication network service provider can analyze the root cause on chronic performance issues and recommend potential solutions is disclosed. The system runs performance analysis on each AAV (mobile backhaul) or other transport networks and collects performance related metrics data. The system then selects a subset of the data related to certain key performance indicators (KPIs), such as latency, jitter, packet loss ratio, and availability. On this subset of KPI-related data, the system applies clustering techniques to identify clusters with similar performance issues. For each cluster, the system binds the AAV performance KPI data with one or more of the following site features—health, location, vendor, market, service type, etc.—to create a cluster map. The system can then generate inferences on root causes of the performance issues.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/669,406, filed on Oct. 30, 2019, entitled NETWORK FAULT DETECTION ANDQUALITY OF SERVICE IMPROVEMENT SYSTEMS AND METHODS, which is herebyincorporated by reference in its entirety.

BACKGROUND

To support end-to-end connectivity, telecommunication network serviceproviders (e.g., ATT, Verizon, T-Mobile, Sprint, etc.) rely on one ormore carrier networks/vendors (alternative access networks, AAV) toprovide backhaul support and connectivity. Since these carrier networksfunction like black-boxes from telecommunication network serviceproviders perspective, it is difficult for the telecommunication networkservice provider to detect, identify, and troubleshoot connectivityand/or network performance issues occurring as a result of thesevendors. For example, it is very difficult for a telecommunicationnetwork service provider to do troubleshooting with the carrier networkvendors because these vendors usually do not agree with thetelecommunication network service providers performance data. And noroot cause analysis is available to the telecommunication networkservice providers on AAV performance issues and outage. As a result, thetelecommunication network service provider is unable to provide theoptimum level of service and performance to its customers.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A-1B are block diagrams illustrating suitable environments withinwhich a Quality of Service (QoS) improvement system operates.

FIG. 2 is a block diagram illustrating the components of the QoSimprovement system.

FIG. 3 is a flow diagram illustrating an example process of transportnetwork fault detection and quality of service (QoS) improvement.

FIGS. 4A-4C are example charts illustrating information of various keyperformance indicators.

FIGS. 5A-5I are example charts illustrating results of applyingclustering models on information of various key performance indicators.

FIGS. 6A-7C are example reports that can be used to identify/analyzenetwork fault trends.

In the drawings, some components and/or operations can be separated intodifferent blocks or combined into a single block for discussion of someof the implementations of the present technology. Moreover, while thetechnology is amenable to various modifications and alternative forms,specific implementations have been shown by way of example in thedrawings and are described in detail below. The intention, however, isnot to limit the technology to the specific implementations described.On the contrary, the technology is intended to cover all modifications,equivalents, and alternatives falling within the scope of the technologyas defined by the appended claims.

DETAILED DESCRIPTION

To solve the above and other problems, the inventors have devised aQuality of Service (QoS) improvement system and method for transportnetwork fault detection and quality of service (QoS) improvement so thata telecommunication network service provider can analyze the root causeon chronic performance issues and recommend potential solutions. Forinstance, the QoS improvement system can help identify whether atransport network fault is occurring due to AAV issues,telecommunication network service provider issues, or any combination ofthe two. The system runs performance analysis on each AAV (mobilebackhaul) and collects performance related metrics data. The system thenselects a subset of the data related to certain key performanceindicators (KPIs), such as latency, jitter, packet loss ratio, andavailability. On this subset of KPI-related data, the system appliesclustering techniques to identify clusters with similar performanceissues. For each cluster, the system binds the AAV performance KPI datawith one or more of the following site features—health, location,vendor, market, etc.—to create a cluster map. For example, the systemcreates a cluster map with location feature to see similarities of theirservice areas. The system can then map the updated clusters with time,vendor, and/or market to identify chronic performance patterns. Usingthis information, the system can then generate inferences on root causesof the performance issues (which are likely caused by vendor networkissues). For example, if a group of nearby sites share similarperformance issues over time, then there is a higher probability thatsomething negative happened in the vendor-shared equipment or links. Inthis manner, the method enables improvement in automation, efficiency,and accuracy of network fault detection. For instance, by providinggreater visibility into the causes of network fault, the system enablestelecommunications service providers to: enforce performance servicelevel agreements (SLAs) with AAVs (e.g., resulting in costs savings inthe form of performance credits from SLAs), accurately identify sourcesand/or locations of faults for efficient remediation (vendor performanceinvestigation and/or improvement, root cause analysis, etc.), improveperformance and troubleshooting efficiency (e.g., from multiple month toa matter of days/hours), avoid network outage with timely maintenance(reliability), track vendor issues, and so on.

In the following description, for the purposes of explanation, numerousspecific details are set forth in order to provide a thoroughunderstanding of implementations of the present technology. It will beapparent, however, to one skilled in the art that implementations of thepresent technology can be practiced without some of these specificdetails.

The phrases “in some implementations,” “in several implementations,”“according to some implementations,” “in the implementations shown,” “inother implementations,” and the like generally mean the specificfeature, structure, or characteristic following the phrase is includedin at least one implementation of the present technology and can beincluded in more than one implementation. In addition, such phrases donot necessarily refer to the same implementations or differentimplementations.

SUITABLE ENVIRONMENTS

FIGS. 1A-1B are block diagrams illustrating suitable environments withinwhich a Quality of Service (QoS) improvement system operates.

One or more user devices 110, such as mobile devices or user equipment(UE) associated with users (such as mobile phones (e.g., smartphones),tablet computers, laptops, and so on), Internet of Things (IoT) devices,devices with sensors, and so on, receive and transmit data, streamcontent, and/or perform other communications or receive services over analternative access vendor (AAV) network 130 and a carrier core network135, which are accessed by the user device 110 over one or more cellsites 120 a-120 n via cell switch routers (122 a). For example, the userdevice 110 can access a carrier core network 135 via a cell site 120 aat a geographical location that includes the cell site, in order totransmit and receive data (e.g., stream or upload multimedia content)from various entities, such as a content provider/data center 140,Internet/cloud data repository 145, and/or other user devices 155 on thenetwork 135. The AAV network 130 is a third-party carrier networkconnecting cell sites 120 a-120 n of a carrier (e.g., Verizon, ATT,T-Mobile, etc.) with the carriers core network 135.

The cell sites 120 a-120 n can include macro cell sites, such as basestations, small cell sites, such as picocells, microcells, orfemtocells, and/or other network access component or sites (includingIEEE 802.11 WLAN access points). The cell cites 120 a-120 n can storedata associated with their operations, including data associated withthe number and types of connected users, data associated with theprovision and/or utilization of a spectrum, radio band, frequencychannel, and so on, provided by the cell sites, and so on. The cellsites 120 a-120 n can monitor their use, such as the provisioning orutilization of PRBs provided by a cell site physical layer in LTEnetwork. For example, a cell site 120 having a channel bandwidth of 5MHz that provides 25 available physical resource blocks through whichdata can be transmitted to/from the user device 110.

Other components provided by the carrier core network 135 can monitorand/or measure the operations and transmission characteristics of thecell sites 120 a-120 n and other network access components. For example,the carrier core network 135 can provide a network monitoring system,via a network resource controller (NRC) or network performance andmonitoring controller, or other network control component, in order tomeasure and/or obtain the data associated with the utilization of cellsites 120 a-120 n when data is transmitted within a telecommunicationsnetwork.

The computing environment 100 includes a QoS improvement system 150configured to monitor aspects of the networks 130 and 135 based on, forexample, data accessed/received from the performance data system 156.The performance data system 156 can measure values for various metrics(including, for example, key performance metrics, such as jitter, frameloss ratio, availability, etc.) at the AAV network 130, carrier corenetwork 135, cell sites (e.g., cell sites 120 a-120 n), etc. to evaluateand select optimum network fault remediation actions to be deployed atvarious points in the communication pathway (e.g., at the AAV network130) to improve network performance as described in detail below. Theperformance data system 156 can measure values using a performanceserver 156 a, which is in communication with probes (e.g., a Sprientprobe) 138 in the carrier core network 135, and store theperformance-related measurement values (performance measurement datarecords) in database 156 b. The AAV performance-related measurementsvalues can be gathered using one or more of the following: TWAMP PM(performance measurement with two-way active measurement protocol(TWAMP) and Sprient system), in real-time (e.g., five packets for fourclass-of-service (e.g., voice, data, control, and management) per secondper Enhanced Video Connection (EVC), measurements by Sprient probes ineach Mobile Switch Office (MSO) trigger tests, measurements usingcross-layer pipeline (e.g., life-cycle measurement, data collection,storage, analytics, and reporting), and so on). The performancemeasurement data records can comprise information related to anassociated location (e.g., where the record was generated), associatedAAV (e.g., AAV vendor name, identifier, location, SLA, etc.), associatedmarket, site identifier, circuit identifier (e.g., identifier whichspecifies the EVC circuit connecting cellular site and MSO over vendornetwork), IP address, port number, measurement protocol like TWAMP,longitude and latitude of the site, market, region, measurement startingtime, measurement ending time, measurement granularity, averageround-trip latency, average round-trip jitter, frame loss ratio (FLR),maximum latency, maximum jitter, maximum FLR, and so on.

AAV network's 130 performance is critical in user experience and networkquality due to its “bridge” functionality. Thus, it is imperative toidentify where a transport network fault is hidden by measuring andevaluating performance data. For example, it is beneficial to identifywhether the network fault is present in the AAV network 130 or in thecarrier core network 135. Such root cause identification and analysisfor chronic performance issues can enable improvement in user experienceand network quality.

FIG. 1 and the discussion herein provide a brief, general description ofa suitable computing environment 100 in which the QoS improvement system150 can be supported and implemented. Although not required, aspects ofthe QoS improvement system 150 are described in the general context ofcomputer-executable instructions, such as routines executed by acomputer, e.g., mobile device, a server computer, or personal computer.The system can be practiced with other communications, data processing,or computer system configurations, including: Internet appliances,hand-held devices (including tablet computers and/or personal digitalassistants (PDAs)), Internet of Things (loT) devices, all manner ofcellular or mobile phones, multi-processor systems, microprocessor-basedor programmable consumer electronics, set-top boxes, network PCs,mini-computers, mainframe computers, and the like. Indeed, the terms“computer,” “host,” and “host computer,” and “mobile device” and“handset” are generally used interchangeably herein, and refer to any ofthe above devices and systems, as well as any data processor.

Aspects of the system can be embodied in a special purpose computingdevice or data processor that is specifically programmed, configured, orconstructed to perform one or more of the computer-executableinstructions explained in detail herein. Aspects of the system can alsobe practiced in distributed computing environments where tasks ormodules are performed by remote processing devices, which are linkedthrough any communications network, such as a Local Area Network (LAN),Wide Area Network (WAN), or the Internet. In a distributed computingenvironment, program modules can be located in both local and remotememory storage devices.

Aspects of the system can be stored or distributed on computer-readablemedia (e.g., physical and/or tangible non-transitory computer-readablestorage media), including magnetically or optically readable computerdiscs, hard-wired or preprogrammed chips (e.g., EEPROM semiconductorchips), nanotechnology memory, or other data storage media. Indeed,computer implemented instructions, data structures, screen displays, andother data under aspects of the system can be distributed over theInternet or over other networks (including wireless networks), on apropagated signal on a propagation medium (e.g., an electromagneticwave(s), a sound wave, etc.) over a period of time, or they can beprovided on any analog or digital network (packet switched, circuitswitched, or other scheme). Portions of the system reside on a servercomputer, while corresponding portions reside on a client computer suchas a mobile or portable device, and thus, while certain hardwareplatforms are described herein, aspects of the system are equallyapplicable to nodes on a network. In an alternative implementation, themobile device or portable device can represent the server portion, whilethe server can represent the client portion.

In some implementations, the user device 110 and/or the cell sites 120a-120 n can include network communication components that enable thedevices to communicate with remote servers or other portable electronicdevices by transmitting and receiving wireless signals using a licensed,semi-licensed, or unlicensed spectrum over communications network, suchas networks 130 and 135. In some cases, the communication networks 130and/or 135 can be comprised of multiple networks, even multipleheterogeneous networks, such as one or more border networks, voicenetworks, broadband networks, service provider networks, InternetService Provider (ISP) networks, and/or Public Switched TelephoneNetworks (PSTNs), interconnected via gateways operable to facilitatecommunications between and among the various networks. Thetelecommunications networks 130 and/or 135 can also include third-partycommunications networks such as a Global System for Mobile (GSM) mobilecommunications network, a code/time division multiple access (CDMA/TDMA)mobile communications network, a 3rd or 4th generation (3G/4G) mobilecommunications network (e.g., General Packet Radio Service(GPRS/EGPRS)), Enhanced Data rates for GSM Evolution (EDGE), UniversalMobile Telecommunications System (UMTS), or Long Term Evolution (LTE)network, 5G mobile communications network, IEEE 802.11 (WiFi), or othercommunications networks. Thus, the user device is configured to operateand switch among multiple frequency bands for receiving and/ortransmitting data.

Further details regarding the operation and implementation of the QoSimprovement system 150 will now be described.

EXAMPLES OF TRANSPORT NETWORK FAULT DETECTION AND QUALITY OF SERVICE(QOS) IMPROVEMENT SOLUTIONS

FIG. 2 is a block diagram illustrating the components of the QoSimprovement system 150. The QoS improvement system 150 can includefunctional modules that are implemented with a combination of software(e.g., executable instructions, or computer code) and hardware (e.g., atleast a memory and processor). Accordingly, as used herein, in someexamples a module is a processor-implemented module or set of code, andrepresents a computing device having a processor that is at leasttemporarily configured and/or programmed by executable instructionsstored in memory to perform one or more of the specific functionsdescribed herein. For example, the QoS improvement system 150 caninclude a KPIs selection module 210, a performance data collectionmodule 220, a clustering module 230, a hidden fault detection module240, a performance improvement module 250, and a reporting andvisualization module 260, each of which is discussed separately below.

THE KPIs SELECTION MODULE

The KPIs selection module 210 is configured and/or programmed to selecta subset of KPIs from a set of Performance Indicators (PIs). The set ofPIs comprises hundreds (for example 200-300) of performance indicators,each of which can be used to measure an aspect of performance of aspecific AAV. For example, the set of PIs can include some or all of thefollowing performance indicators: delay, jitter, frame loss ratio,availability, traffic intensity, number of users, dropped call rate, PRButilization, CQI, throughput, bandwidth, bandwidth utilization, carrieraggregation, advanced Quadrature Amplitude Modulation (QAM) efficiency,device health such as CPU utilization, memory utilization, availablememory, I/O dropped frame percentage, system alarms, MTTR, MTTF,alternate/available AAV, population served by the AAV, households servedby the AAV, and so on.

From this set of numerous performance indicators, the KPIs selectionmodule 210 selects a subset of key performance indicators to be used toevaluate and compare impact/performance of various network performanceimprovement solutions. The KPIs selection module 210 selects the subsetof key performance indicators based on one or more of the followingfactors: correlation of each performance indicator with networkperformance, correlation of each performance indicator with customerexperience, correlation of each performance indicator with otherperformance indicators, user (for example, administrator) preference,telecommunications service provider preference, and so on. For instance,the KPIs selection module 210 selects performance indicators whichexhibit a low degree of correlation yet reflect the dimensions of theoverall composite. For example, the KPIs selection module 210 selectsthe following key performance indicators as components of the subset ofkey performance indicators: delay, jitter, frame loss ratio, andavailability. Alternatively or additionally, the KPIs selection module210 selects a top threshold number of performance indicators having amaximum correlation with the customer experience. In someimplementations, the KPIs selection module 210 selects components of thesubset of key performance indicators based on principal componentanalysis, bandwidth utilization, alarm numbers, device health statisticssuch as CPU utilization, memory utilization, I/O utilization/busy timelength, and so on.

THE PERFORMANCE DATA COLLECTION MODULE

The performance data collection module 220 is configured and/orprogrammed to receive data from one or more sources regarding values ofthe performance indicators in the subset of key performance indicatorsand/or the set of performance indicators. The performance datacollection module 220 can retrieve performance data (performancemeasurement data records or AAV performance KPI data) maintained/storedby the performance data system (FIG. 1B, 156) and/or the QoS improvementdatabase 255. The performance measurement data records can compriseinformation related to an associated location (e.g., where the recordwas generated), associated AAV, associated market, and so on. Theperformance data collection module 220 can retrieve data at a certaingranularity (e.g., daily, weekly, monthly, quarterly, yearly, etc.)based on one or more of the following factors: AAV identity, market,service type, location, SLA, and so on. For example, for a first AAV,the performance data collection module 220 retrieves monthly data foranalysis, whereas for a second (more reliable AAV), the performance datacollection module 220 retrieves semi-annual data. In severalimplementations, the performance data collection module 220 retrievesweekly, daily or even hourly data to provide more granularity.

FIGS. 4A-4C are example charts illustrating performance data of variouskey performance indicators. FIG. 4A illustrates chart 405 that shows thevalues of KPI average delay associated with various markets (e.g., SanFrancisco, San Diego, Washington D.C., and so on). FIG. 4B illustratesreport 410 a that shows values of four KPIs—average delay, averagejitter, packet loss ratio, and availability—associated with various AAVvendors (e.g., Lightower, Fibertech, ATT, and so on). FIG. 4B furtherillustrates chart 410 b that shows the values of KPI average delayassociated with various vendors. FIG. 4C illustrates chart 415 thatshows the values of KPI frame loss ratio associated with variousvendors. The KPI values can differ from vendor to vendor, and market tomarket. In several implementations, the performance data collectionmodule 220 retrieves performance data for every time period t (e.g.,every week, month, quarter, year, and so on).

In several implementations, the performance data collection module 220can collect the values of the performance indicators before a particularnetwork fault remediation action/solution is deployed (pre-solutiondeployment), after the particular network fault remediationaction/solution is deployed (post-solution deployment), or both.

The performance data collection module 220 collects/receives/accessesone or more of the following data records associated with theperformance indicators (which can be stored in the QoS improvementdatabase 255): location specific records (LSR), call data records(CDRs), timing advance values, RF signal data, distance between thecustomer and at least one telecommunications network site, strength ofsignal, quantity of data used, type of device of the customer,applications data (e.g., application type, name, owner, manager, datasent/received/used/saved, bandwidth used, APIs accessed, etc.), sourceof usage records (for example, telecommunications service provider,third-party, application owner, etc.). Examples of other types of datacollected by the performance data collection module 220 include, but arenot limited to, data collected from third party applications (e.g.,including crowdsourced data) that can help to determine customerexperience with location. For example, the performance data collectionmodule 220 can collect information of a users location using his/hersocial media posts (e.g., tweets, check-ins, posts, etc.). As anotherexample, the performance data collection module 220 collects applicationlevel data (e.g., collected using applications related to Internet ofThings (loT) devices, sensors, billing meters, traffic lights, etc.) toidentify the user location and/or data related to the performanceindicators.

THE CLUSTERING MODULE

The clustering module 230 is configured and/or programmed to applyclustering techniques to detect clusters with similar performanceissues. The clustering module 230 can apply k-means clustering or otherclustering machine learning algorithm(s) to divide the performancevalues of the selected KPIs into different groups based on theirperformance similarity distance (e.g., Euclidian distance). In severalimplementations, the clustering module 230 applies techniques such asthe elbow method, silhouette analysis, etc. to determine an optimalvalue of k-number in k-means clustering, which help in determining finalcluster number for the performance data. After generating the clusters,the clustering module 230 can bind the AAV performance KPI data with oneor more of the following site/store/office features—health, location,vendor, market, etc.—to create a cluster map, which can be used todetermine if there are multiple sites/stores/offices nearby with similarperformance pattern and in a same performance cluster. For example, theclustering module 230 creates a cluster map with location feature to seesimilarities of their service areas. The clustering module 230 can thenmap the updated clusters with time, vendor, and/or market to identifychronic performance patterns.

FIGS. 5A-5I are example charts illustrating results of applyingclustering models on information of various key performance indicators.The clustering module 230 can first run performance analysis on AAVperformance KPI data, as illustrated in FIG. 5A (a chart 505 depictingnormalized performance distribution). Using the performance analysisresults, the clustering module can determine the distribution of the AAVperformance KPI data. For example, for the normalized performancedistribution illustrated in FIG. 5A, the clustering module 230determines that the AAV performance KPI data (latency, jitter, and frameloss ratio) are unevenly distributed, and that many sites suffer frombad performance.

The clustering module 230 then applies data-analytics techniques withk-means clustering to identify an optimal k-value (using, e.g., theelbow method, the silhouette method, etc.). For example, as illustratedin FIG. 5B, the clustering module 230 depicts a chart 510 thatidentifies an optimal k-value as four (4) for the performance dataillustrated in FIG. 5A. Each cluster features are differentiated by itscentroid KPI values, for example as shown in table 520 of FIG. 5D.

After creating the optimum number of clusters, the clustering moduleevaluates features of each cluster. For example, the clustering modulecan determine that a first cluster depicts similar and degraded frameloss ratio, while another cluster depicts similar and degraded latency,and so on. The clustering module then maps performance clusters to oneor more clustering parameters (e.g., vendor, market, service level, sitelocations, health information, time, etc.) to identify sites that havesimilar performance issues. For example, as depicted in chart 515 ofFIG. 5C, the clustering module updates/maps each cluster with sitelocation information to determine similarity of service areas associatedwith each cluster, and generate a cluster map with locations. FIG. 5Eillustrates tables 530 a, 530 b, and 530 c for different clusters mappedto vendors and sites, which illustrate distribution for problem siteswith vendors. FIGS. 5F and 5G illustrate charts 540 a and 540 brespectively, which illustrate that different vendors have differentperformance issues in different serving areas (e.g., some performanceclusters are dominant in a certain vendor network and area). Thesefigures show the correlation among clusters, vendors, site locations,and service levels.

HIDDEN FAULT DETECTION MODULE

The hidden fault detection module 240 is configured and/or programmed togenerate inferences on root causes of the performance issues (which arelikely caused by vendor network issues) using the cluster maps. If agroup of nearby sites share similar performance issues over time, thenthere is a higher probability that something negative happened in theAAV-shared equipment or links. For example, the hidden fault detectionmodule 240 can analyze chart 550 illustrated in FIG. 5H to determinethat certain sites have similar performance issues in certain areas, andthat certain AAVs have chronic performance issues in certain areas andservice type. As another example, for sites in the same area/location,which are marked with same cluster features and that have similarperformance problem over time (month to month, week to week, day to day,or other period length), the hidden fault detection module 240 canprovide insights that vendor subnet or vendor device which provideconnections to the group of sites have high probability ofresponsibility for end-to-end performance degradation.

As another example, as illustrated in FIGS. 6A-6B, the hidden faultdetection module 240 can analyze the inter-site distances (site distancedistribution) in same cluster (AAV vendor cluster 3) to infer with ahigh probability that the vendor network (device or link) is responsiblefor degradation of a telecommunication network service provider'sperformance. The hidden fault detection module 240 can perform periodicperformance cluster tracking to identify network fault trends (e.g.,vendor service consistency, reliability, potential outage if clustersize grows, etc.) in the telecommunications service network. Forexample, as illustrated in FIGS. 7A-7C, the hidden fault detectionmodule 240 can perform periodic performance cluster tracking over threedifferent time periods for the same market to identify similar problemsin same area with same vendors for a group of sites over time.Similarly, as illustrated in FIG. 6D, the hidden fault detection module240 can perform periodic performance cluster tracking over two differenttime periods (one month apart) for the same market (e.g., Milwaukee) toidentify similar problems (bad frame loss ratio) in same area with samevendor for a group of sites over time.

PERFORMANCE IMPROVEMENT MODULE

The performance improvement module 250 is configured and/or programmedto use the results of the clustering module and the hidden faultdetection module to identify and/or perform network fault remediationactions (e.g., more efficient troubleshooting with the AAVs, perform AAVcomparison with detailed market, area, and locations, run strategicvendor selection for better vendor service, and so on). Other examplesof fault remediation actions include, but are not limited to thefollowing. For example, operation teams can regularly monitor theresults from the hidden fault detection module 240 to identify chronicperformance area and related vendor network so that they can issuetickets to the vendor for troubleshooting instead of manualtroubleshooting site by site by themselves. As another example, a vendorperformance management team can the results from the hidden faultdetection module 240 as the basis for discussions with vendors forperformance SLA enforcement with much stronger evidence to show relatedvendor poor performance and responsibility. Even without the vendor'sown data, telecommunication network service providers can doapple-to-apple comparison by comparing multiple vendor performanceresults in same area (which results in more accountability by thevendor). The team can also use the results to choose the best vendorservice based on their performance trend and pattern, and then use theirnetwork as AAV access to improve site performance. As another example,once the system identifies that more and more sites served by samevendor in a specific area get worse performance over time, the systemcan then interpret that as a sign that in short future the vendornetwork may get down or suffer an outage. So the system providesprediction on the potential outage and the telecommunication networkservice provider's operations team can proactively work with the vendorto do high-priority troubleshooting or maintenance to avoid thepotential outage disaster.

In several implementations, the performance improvement module 250identifies more than one network fault remediation actions/solutions,ranks the identified actions/solutions and then select one or moreactions/solutions as candidates for deployment based on one or more ofthe following factors: agreements with the AAVs, severity of performancedegradation, KPI being evaluated, market, vendor, time, location,service level information, cost of deploying the solution, and so on. Insome implementations, the ranking factor can be set based on differentteam job requirements and scope. For example, operation team can sethigher priority on severity of performance degradation and focus onsites with the worst performance and do troubleshooting with a group ofsites based on the system.

REPORTING AND VISUALIZATION MODULE

The reporting and visualization module 260 is configured and/orprogrammed to provide reports (ad hoc, pre-built, etc.) and analysis ofnetwork fault trends. FIGS. 6A-7C are example reports that can beprovided by the reporting and visualization module 260.

FLOW DIAGRAMS

FIG. 3 is a flow diagram illustrating a process 300 of transport networkfault detection and quality of service (QoS) improvement. Process 300begins at block 305 where it receives/retrieves/accesses performancemeasurement data records (AAV performance-related measurements values).The performance measurement data records comprise information about anassociated location, an associated vendor, an associated market, and soon. At block 310, process 300 selects key performance indicators (KPIs).For example, as discussed above, process 300 selects a subset of keyperformance indicators from a set of performance indicators based on acorrelation of each performance indicator with network performance. Atblock 315, process 300 generates a set of clusters of the set ofperformance measurement data records using values corresponding to keyperformance indicators in the subset of key performance indicators. Therecords in each cluster can have similar values for the selected keyperformance indicators. At block 320, process 300 maps the clusters withat least one clustering parameter (e.g., vendor information, locationinformation, market information, service level information, etc.) togenerate mapped updated cluster records. For example, process 300 canfirst update each cluster with site location information to determinesimilarity of service areas associated with each cluster, and then mapthe updated clusters with additional clustering parameters. At block325, process 300 identifies one or more network fault trends (e.g.,sites in certain markets of same vendor suffer from similar performanceissues). At block 330, process 300 generates and/or executes networkfault remediation solutions/actions to minimize the identified networkfaults. At block 335, process 300 generates customized reports and/orvisualizations (e.g., illustrated in FIGS. 6A-7C) for the analysis.

CONCLUSION

Unless the context clearly requires otherwise, throughout thedescription and the claims, the words “comprise,” “comprising,” and thelike are to be construed in an inclusive sense, as opposed to anexclusive or exhaustive sense; that is to say, in the sense of“including, but not limited to.” As used herein, the terms “connected,”“coupled,” or any variant thereof, means any connection or coupling,either direct or indirect, between two or more elements; the coupling ofconnection between the elements can be physical, logical, or acombination thereof. Additionally, the words “herein,” “above,” “below,”and words of similar import, when used in this application, shall referto this application as a whole and not to any particular portions ofthis application. Where the context permits, words in the above DetailedDescription using the singular or plural number can also include theplural or singular number respectively. The word “or,” in reference to alist of two or more items, covers all of the following interpretationsof the word: any of the items in the list, all of the items in the list,and any combination of the items in the list.

The above detailed description of implementations of the system is notintended to be exhaustive or to limit the system to the precise formdisclosed above. While specific implementations of, and examples for,the system are described above for illustrative purposes, variousequivalent modifications are possible within the scope of the system, asthose skilled in the relevant art will recognize. For example, somenetwork elements are described herein as performing certain functions.Those functions could be performed by other elements in the same ordiffering networks, which could reduce the number of network elements.Alternatively, or additionally, network elements performing thosefunctions could be replaced by two or more elements to perform portionsof those functions. In addition, while processes, message/data flows, orblocks are presented in a given order, alternative implementations canperform routines having blocks, or employ systems having blocks, in adifferent order, and some processes or blocks can be deleted, moved,added, subdivided, combined, and/or modified to provide alternative orsubcombinations. Each of these processes, message/data flows, or blockscan be implemented in a variety of different ways. Also, while processesor blocks are at times shown as being performed in series, theseprocesses or blocks can instead be performed in parallel, or can beperformed at different times. Further, any specific numbers noted hereinare only examples: alternative implementations can employ differingvalues or ranges.

The teachings of the methods and system provided herein can be appliedto other systems, not necessarily the system described above. Theelements, blocks and acts of the various implementations described abovecan be combined to provide further implementations.

Any patents and applications and other references noted above, includingany that can be listed in accompanying filing papers, are incorporatedherein by reference. Aspects of the technology can be modified, ifnecessary, to employ the systems, functions, and concepts of the variousreferences described above to provide yet further implementations of thetechnology.

These and other changes can be made to the invention in light of theabove Detailed Description. While the above description describescertain implementations of the technology, and describes the best modecontemplated, no matter how detailed the above appears in text, theinvention can be practiced in many ways. Details of the system can varyconsiderably in its implementation details, while still beingencompassed by the technology disclosed herein. As noted above,particular terminology used when describing certain features or aspectsof the technology should not be taken to imply that the terminology isbeing redefined herein to be restricted to any specific characteristics,features, or aspects of the technology with which that terminology isassociated. In general, the terms used in the following claims shouldnot be construed to limit the invention to the specific implementationsdisclosed in the specification, unless the above Detailed Descriptionsection explicitly defines such terms. Accordingly, the actual scope ofthe invention encompasses not only the disclosed implementations, butalso all equivalent ways of practicing or implementing the inventionunder the claims.

While certain aspects of the technology are presented below in certainclaim forms, the inventors contemplate the various aspects of thetechnology in any number of claim forms. For example, while only oneaspect of the invention is recited as implemented in a computer-readablemedium, other aspects can likewise be implemented in a computer-readablemedium. Accordingly, the inventors reserve the right to add additionalclaims after filing the application to pursue such additional claimforms for other aspects of the technology.

1. A computer-implemented method for detecting faults in atelecommunications service network, the method comprising: receiving aset of performance measurement data records measuring performance of atleast one network access component of a vendor; selecting a subset ofkey performance indicators from a set of performance indicators based ona correlation of each of the set of performance indicators with anetwork performance; generating a set of clusters of the set ofperformance measurement data records using values corresponding to keyperformance indicators in the subset of key performance indicators;updating each cluster in the set of clusters with site locationinformation to determine similarity of service areas associated witheach cluster of the set of clusters; and mapping the updated clusters inthe set of clusters with at least one clustering parameter to generatemapped updated cluster records.
 2. The method of claim 1, wherein theset of clusters is generated using k-means clustering, and wherein anoptimal k value is determined using an elbow methodology.
 3. The methodof claim 1, wherein the set of clusters is generated using k-meansclustering, and wherein an optimal k value is determined using asilhouette methodology.
 4. The method of claim 1, further comprising:receiving the set of performance measurement data records for every timeperiod t, generating the mapped updated cluster records for each of theset of performance measurement data records received for each timeperiod t, and analyzing the mapped updated cluster records for each ofthe set of performance measurement data records received for each timeperiod t to identify at least one network fault trend in thetelecommunications service network.
 5. The method of claim 1, furthercomprising: identifying at least one network fault occurring in thetelecommunications service network using the generated mapped updatedcluster records.
 6. The method of claim 1, further comprising:identifying at least one network fault remediation action capable ofbeing performed in the telecommunications service network based on thegenerated mapped updated cluster records.
 7. The method of claim 1,wherein the subset of key performance indicators comprises one or moreof: latency, jitter, packet loss ratio, or availability.
 8. At least onenon-transitory computer-readable medium containing instructions, thatwhen executed by a processor, performs operations for detecting faultsin a telecommunications service network, the operations comprising:receiving a set of performance measurement data records measuringperformance of at least one network access component of a vendor;selecting a subset of key performance indicators from a set ofperformance indicators based on a correlation of each of the set ofperformance indicators with a network performance; generating a set ofclusters of the set of performance measurement data records using valuescorresponding to key performance indicators in the subset of keyperformance indicators; updating each cluster in the set of clusterswith site location information to determine a similarity of serviceareas associated with each cluster of the set of clusters; and mappingthe updated clusters in the set of clusters with at least one clusteringparameter to generate mapped updated cluster of the records.
 9. The atleast one non-transitory one computer-readable medium of claim 8,wherein the set of clusters is generated using k-means clustering, andwherein an optimal k value is determined using an elbow methodology. 10.The at least one non-transitory one computer-readable medium of claim 8,wherein the set of clusters is generated using k-means clustering, andwherein an optimal k value is determined using a silhouette methodology.11. The at least one non-transitory one computer-readable medium ofclaim 8, wherein the operations further comprise: receiving the set ofperformance measurement data records for every time period t, generatingthe mapped updated cluster records for each of the set of performancemeasurement data records received for each time period t, and analyzingthe mapped updated cluster records for each of the set of performancemeasurement data records received for each time period t to identify atleast one network fault trend in the telecommunications service network.12. The at least one non-transitory one computer-readable medium ofclaim 8, wherein the operations further comprise: identifying at leastone network fault occurring in the telecommunications service networkusing the generated mapped updated cluster records.
 13. The at least onenon-transitory one computer-readable medium of claim 8, wherein theoperations further comprise: identifying at least one network faultremediation action capable of being performed in the telecommunicationsservice network based on the generated mapped updated cluster records.14. The at least one non-transitory one computer-readable medium ofclaim 8, wherein the subset of key performance indicators comprises oneor more of: latency, jitter, packet loss ratio, or availability.
 15. Asystem for detecting faults in a telecommunications service networkcomprising at least one non-transitory computer-readable medium havinginstructions stored thereon, which when executed by one or moreprocessors of the system cause the system to: receive a set ofperformance measurement data records measuring performance of at leastone network access component of a vendor; select a subset of keyperformance indicators from a set of performance indicators based on acorrelation of each of the set of performance indicators with networkperformance; generate a set of clusters of the set of performancemeasurement data records using values corresponding to one or more keyperformance indicators in the subset of key performance indicators;update each cluster in the set of clusters with a site locationinformation to determine a similarity of service areas associated witheach cluster of the set of clusters; and map the updated clusters in theset of clusters with at least one clustering parameter to generatemapped updated cluster of the records.
 16. The system of claim 15,wherein the set of clusters is generated using k-means clustering, andwherein an optimal k value is determined using an elbow methodology or asilhouette methodology.
 17. The system of claim 15, wherein theinstructions when executed by the one or more processors of the systemfurther cause the system to: receive the set of performance measurementdata records for every time period t, generate the mapped updatedcluster records for each of set of performance measurement data recordsreceived for each time period t, and analyze the mapped updated clusterrecords for each of the set of performance measurement data recordsreceived for each time period t to identify at least one network faulttrend in the telecommunications service network.
 18. The system of claim15, wherein the instructions when executed by the one or more processorsof the system further cause the system to: identify at least one networkfault occurring in the telecommunications service network using thegenerated mapped updated cluster records.
 19. The system of claim 15,wherein the instructions when executed by the one or more processors ofthe system further cause the system to: identifying at least one networkfault remediation action capable of being performed in thetelecommunications service network based on the generated mapped updatedcluster of the records.
 20. The system of claim 15, wherein the subsetof key performance indicators comprises one or more of: latency, jitter,packet loss ratio, availability.